Run Kimi K2.6 NVFP4 Dynamo vLLM on b200-new / 在 b200-new 上运行 Kimi K2.6 NVFP4 Dynamo vLLM - #2438
Run Kimi K2.6 NVFP4 Dynamo vLLM on b200-new / 在 b200-new 上运行 Kimi K2.6 NVFP4 Dynamo vLLM#2438RohitNagraj wants to merge 8 commits into
Conversation
中文：将 Kimi K2.6 B200 Dynamo vLLM 扫描切换到 b200-new，并移除 DEP4 到 DEP8、并发 1024 的配置项。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：更新性能变更日志中的拉取请求链接。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM aside from the inline nit (unfilled pr-link placeholder).
Extended reasoning...
Overview
This PR is a routing/config change for the kimik2.6-fp4-b200-dynamo-vllm benchmark: it switches the runner pool from b200-multinode to a new b200-new pool (added to configs/runners.yaml with 9 nscale-slurm node labels), removes a now-unused 1P/1D DEP4-to-DEP8 concurrency-1024 sweep entry and its corresponding recipe YAML, adds a changelog entry, and introduces a thin new launcher script runners/launch_b200-nscale-slurm.sh that simply execs the existing launch_b200-dgxc.sh with the same arguments.
Security risks
None. This is benchmark infrastructure/config only — no auth, crypto, user input handling, or externally-reachable code paths are touched. The new launcher script contains no injection risk since it forwards "" verbatim to a trusted local script.
Level of scrutiny
Low-to-moderate. This follows an established pattern in the repo (adding a new named runner pool that delegates to an existing DGXC launcher, as verified by reading launch_b200-dgxc.sh) and is mechanical in nature — a runner field swap, a config-block removal, and a label list addition. No business logic or shared production code paths are affected; a bad routing choice here would surface immediately as a failing benchmark sweep job rather than silently corrupting data.
Other factors
The only finding from this run's bug hunt is a cosmetic nit (unfilled pr-link: .../pull/XXX placeholder in the append-only perf-changelog.yaml), which does not affect any runtime behavior. A candidate concern about the new launcher reusing DGXC-specific SLURM partition/account and model paths was raised and ruled out — I independently confirmed launch_b200-dgxc.sh exists and this delegation pattern is intentional and consistent with the PR description ("reuse the existing B200 DGXC launcher").
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2213 | ||
|
|
||
| - config-keys: | ||
| - kimik2.6-fp4-b200-dynamo-vllm | ||
| description: | ||
| - "Run the Kimi K2.6 NVFP4 B200 Dynamo vLLM disaggregated configuration on the b200-new runner." | ||
| - "Remove the 1P/1D DEP4-to-DEP8 concurrency-1024 configuration." | ||
| - "Image: vllm/vllm-openai:v0.25.1" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2438 |
There was a problem hiding this comment.
🟡 The new perf-changelog.yaml entry for kimik2.6-fp4-b200-dynamo-vllm ends with an unfilled placeholder link: pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX instead of the real PR number. Please replace XXX with 2438 to match every other entry in this append-only changelog.
Extended reasoning...
The newly appended block in perf-changelog.yaml (lines 5357-5365) documents the routing change to b200-new, but its pr-link field was never filled in with the actual PR number and instead ships with the literal placeholder https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.
This is verifiable directly against the committed content rather than the rendered PR diff. Running git log --oneline -- perf-changelog.yaml shows commit d6e0efe ("chore: route Kimi K2.6 sweep to b200-new") as the tip of history for this file, with a subject line that matches this PR's title exactly and — unlike the surrounding merged commits ((#2213), (#2360), (#2312)) — carries no merge-PR suffix, confirming it is this PR's own commit rather than a later, unrelated change. Both git show HEAD:perf-changelog.yaml and the on-disk working tree confirm the last line of the new block reads pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.
One verifier raised a plausible-sounding objection: the preloaded PR diff shown in review context displays pull/2438 on that line, and standard guidance says to prefer the diff over a possibly-diverged working tree. However, that guidance exists to handle the case where the checkout is ahead of the PR (later commits touched the file). That doesn't apply here — HEAD is the PR's commit, there is no later commit to discount, and the diff-displayed 2438 does not match what git actually recorded as committed. In other words, the "2438" appears to be a harness-level auto-substitution for display purposes, not the real content of the change. Trusting git show/working-tree content (the actual bytes that would land in the repo) over the rendered diff is correct in this specific case, since they conflict and only one can be what actually gets merged.
Step-by-step proof:
git log --oneline -- perf-changelog.yaml→ top commit isd6e0efe, no(#NNNN)suffix, subject matches this PR's title.git show HEAD:perf-changelog.yaml | tail -9→ last line of the newkimik2.6-fp4-b200-dynamo-vllmblock ispr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX.tail -9 perf-changelog.yaml(working tree) → identical.../pull/XXXline, confirming it is not a stale artifact of an intermediate git state.- Every other entry in the file (e.g. the immediately preceding one ending
.../pull/2213, and others at.../pull/2360,.../pull/2312) has a real, resolved PR number — this is the only entry left as a placeholder.
Impact is purely cosmetic: perf-changelog.yaml is an append-only audit/documentation log, and this field is not consumed by any routing, sweep, or runtime logic — nothing breaks functionally. But it leaves a dead link (.../pull/XXX 404s) in an otherwise-accurate historical record, which is a real, if minor, correctness gap. The fix is a one-line change: replace XXX with 2438 on line 5365.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30659217395 |
Configure the nscale Slurm account and partition, shared squash cache, model path, and runner temp storage while retaining the common B200 launcher defaults. 中文：配置 nscale 的 Slurm 账户和分区、共享 squash 缓存、模型路径以及 runner 临时存储，同时保留通用 B200 启动器的默认行为。
Use the shared squash cache with enroot local temporary storage and allow the first serialized image import enough time to finish. 中文：使用共享 squash 缓存和 enroot 本地临时存储，并为首次串行镜像导入预留足够的完成时间。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30660787348 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30662010462 |
中文：将 b200-new 启动器的 Slurm 分区从 batch_all 更新为 batch_1。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30668476660 |
中文：将 b200-new 启动器的 Slurm 分区更新为 batch_2。
Restore the b200-new Slurm partition from batch_2 to batch_1.\n\n中文：将 b200-new 的 Slurm 分区从 batch_2 恢复为 batch_1。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30669462128 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30669586535 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30669586535 |
|
/reuse-sweep-run |
kedarpotdar-nv
left a comment
kedarpotdar-nv
left a comment
There was a problem hiding this comment.
Before CODEOWNER sign-off, please register the new nscale runner fleet under a cluster identity in configs/runners.yaml (for example, cluster:b200-nscale-slurm, listing the same runners as b200-new). The runner setup guide requires one cluster:<name> identity per fleet for success-rate reporting.
The exact-head Run Sweep 30669586535, attempt 2 is green: all 6 throughput jobs and all 5 selected eval jobs passed. However, its run-stats artifact contains no nscale bucket, and the calc-success-rate output has only the table header with no hardware rows. This happens because utils/calc_success_rate.py discovers fleets from the cluster:* entries in configs/runners.yaml, while this PR adds only the scheduling alias b200-new.
Please add the cluster-label entry and obtain validation on the updated head. Once that is fixed, I can post the CODEOWNER checklist sign-off.
中文：在 CODEOWNER 签署之前，请在 configs/runners.yaml 中为新的 nscale runner 池注册集群标识（例如 cluster:b200-nscale-slurm，runner 列表与 b200-new 相同）。Runner 配置指南要求每个集群配置一个 cluster:<name> 标识，以便正确统计任务成功率。
精确对应当前提交的 Run Sweep 30669586535（attempt 2） 已全部通过：6 个吞吐量任务和 5 个评估任务均成功。但是，生成的 run-stats 产物中没有 nscale 集群条目，calc-success-rate 输出也只有表头，没有任何硬件数据行。原因是 utils/calc_success_rate.py 通过 configs/runners.yaml 中的 cluster:* 条目识别集群，而本 PR 目前只新增了调度别名 b200-new。
请补充集群标签条目，并在更新后的提交上完成验证。修复后我即可发布 CODEOWNER checklist 签署。
kedarpotdar-nv
left a comment
kedarpotdar-nv
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Run Sweep 30669586535, attempt 2
- Verified that this PR passes evals. Run Sweep 30669586535, attempt 2
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- Scope: multi-node disaggregated STP only; speculative decoding, Agentic golden-AL, and single-node recipe requirements are not applicable.
- Exact-head evidence: Run Sweep 30669586535, attempt 2 ran on
fc2609111631afc2363f1133fad5a960e9638845; all 6 selected throughput jobs and all 5 selected eval jobs passed. GSM8K scores were0.965125–0.971948, above the0.90threshold, using the upstreamvllm/vllm-openai:v0.25.1image. - This PR changes runner routing and launch-environment plumbing and removes one obsolete concurrency configuration. It introduces no FLOP-reducing, speculative-decoding, Agentic, or inference-engine patching changes.
- Accepted non-functional follow-up: the new
b200-nscale-slurm_*fleet is scheduled throughb200-newbut is not yet duplicated under acluster:*identity inconfigs/runners.yaml. As a result, this sweep'srun-statsartifact omits the nscale success-rate row. This does not affect the benchmark/eval jobs or their result artifacts; cluster success-rate registration will be handled separately after this PR merges. - Existing authorized reuse command: #2438 (comment)
Signed: kedarpotdar-nv
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
kedarpotdar-nv
left a comment
kedarpotdar-nv
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Run Sweep 30669586535, attempt 2
- Verified that this PR passes evals. Run Sweep 30669586535, attempt 2
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- Scope: multi-node disaggregated STP only; speculative decoding, Agentic golden-AL, and single-node recipe requirements are not applicable.
- Exact-head evidence: Run Sweep 30669586535, attempt 2 ran on
fc2609111631afc2363f1133fad5a960e9638845; all 6 selected throughput jobs and all 5 selected eval jobs passed. GSM8K scores were0.965125–0.971948, above the0.90threshold, using the upstreamvllm/vllm-openai:v0.25.1image. - This PR changes runner routing and launch-environment plumbing and removes one obsolete concurrency configuration. It introduces no FLOP-reducing, speculative-decoding, Agentic, or inference-engine patching changes.
- Accepted non-functional follow-up: the new
b200-nscale-slurm_*fleet is scheduled throughb200-newbut is not yet duplicated under acluster:*identity inconfigs/runners.yaml. As a result, this sweep'srun-statsartifact omits the nscale success-rate row. This does not affect the benchmark/eval jobs or their result artifacts; cluster success-rate registration will be handled separately after this PR merges. - Existing authorized reuse command: #2438 (comment)
Signed: kedarpotdar-nv
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
overall lgtm, just quick feedback in slack
4 participants
Summary
kimik2.6-fp4-b200-dynamo-vllmto theb200-newrunner pool and reuse the existing B200 DGXC launcher.中文说明
kimik2.6-fp4-b200-dynamo-vllm切换到b200-newrunner 池，并复用现有的 B200 DGXC 启动器。